Weaknesses: - It is not really clear what types of event are available, and what is the difference between events and episodes. This goes with the point above, it would be nice to have a more in-depth description of the events, their frequency and the possibility of predicting them given the input. Is it simply a label attached to the 384 2 km inputs or is it localized within each image, for each time? It seem that the values of weather radars are very skewed towards 0s, and large values very rare. Also, I wonder if maybe there are some more domain specific loss functions to be optimized, eg taking into account spatial smoothness of signals, rarity of levels, level sets of precipitation, etc.

Four knowledgeable reviewers all appreciated the contributions of this paper and rated it as above the bar for publication at NeurIPS. Reviewers acknowledged that the primary contribution was the curation of a dataset and benchmark tasks on the data set, and not novel methods, but felt that the curation of a large, high-quality data set for real tasks in atmospheric/earth sciences is important and could spur AI work in this area. The authors deserve credit for this. Additionally, the reviewers appreciated that the baseline methods developed for the benchmark tasks were themselves thoughtful and significant, if not highly novel from a methods perspective. The reviewers asked a number of questions about justification and details of the data set construction, evaluation metrics, and baselines.

Modern deep learning approaches have shown promising results in meteorological applications like precipitation nowcasting, synthetic radar generation, front detection and several others. In order to effectively train and validate these complex algorithms, large and diverse datasets containing high-resolution imagery are required. Petabytes of weather data, such as from the Geostationary Environmental Satellite System (GOES) and the Next-Generation Radar (NEXRAD) system, are available to the public; however, the size and complexity of these datasets is a hindrance to developing and training deep models. To help address this problem, we introduce the Storm EVent ImagRy (SEVIR) dataset - a single, rich dataset that combines spatially and temporally aligned data from multiple sensors, along with baseline implementations of deep learning models and evaluation metrics, to accelerate new algorithmic innovations. SEVIR is an annotated, curated and spatio-temporally aligned dataset containing over 10,000 weather events that each consist of 384 km x 384 km image sequences spanning 4 hours of time.